Lecture 1

Paper and articles :

- What uncertainties do we need in bayesian learning ? Alex kendall et al 2017 nips
- Computer vision : models, learning and inference : Simon Prince Book
- Pattern Recognition and Machine Learning by Bishop 
- https://medium.com/@jdwittenauer/a-sampling-of-monte-carlo-methods-8a37bfd19
- https://sudeepraja.github.io/Bayes/
- https://www.countbayesie.com
- Kingma and Weiling et al 2014 , Auto encoding variational bayes
- Rezende, Mohammed et al ICML 2014 VAE
- Aaron Couville, DLSS 2015 VAE
- Semi Supervised learning Kingma et al.
- Semi Supervised Learning with VAE , github
- Attribute2Image, ICCV 2015
- Ruslan Salakhutidanov et al VAE ICLR 2016 
- Deep Convolution Inverse graphic network, tejas d kulkarni et al
- Ian goodfellow 2015 et al VAE
- https://towardsdatascience.com/intuitively-understanding-variational-autoencoders-1bfe67eb5daf
- yarin.co/blog 
- Ghahramani, Gal et al 2017 Medical images 
- Yarin gall on Bayesian deep learning
- https://alexgkendall.com/computer_vision/bayesian_deep_learning_for_safe_ai/
- DRAW, ICML 2015
- Show, Attend and tell ICML 2015
- attention-mechanism, heuritech
- Spatial Transformer Network 
- Reccurent Spatial Transformers
- AlignDRAW
- Attend, Infer, Repeat arxiv 2016.
- Video Generation using Dual Attention ICCV 2017
- Two Stage video Generation
- Adversarial data programming CVPR 2018
- Toward Controlled Generation of Text
- https://jaan.io/what-is-variational-autoencoder-vae-tutorial/

Overview :

• Motivation
• Probability theory fundamentals
• Inference with Conjugate priors
• Approximate inference with laplace approximation
• Approximate inference with monte carlo sampling

Motivation

• Not all deep learning models are interpretable, this approach lets us understand what our model knows and what it doesnt know
• Principled framework and semi-supervised learning can incorporated easily
• Can work with small set of samples
• If my method is not doing well, then we can obtain "uncertainty estimates" which helps us get where its not doing well

Probability theory fundamentals

• Random Variable (could be discrete or continuous)
• Marginal, Conditional Probability & Joint probability
• Sum & Product rule
• Through marginalization, Marginal Probabilities can be obtained from joint probability
• Pr(x,y) = pr(x|y)pr(y)
• Bayes rule
• Bayes rule is used to obtain a set of parameters from the input and then the output is obtained by marginalizing over the parameters
• Expectation value & properties

Inference with Conjugate Priors

• The inference problem is to compute posterior distribution i.e distribution of theta.
• Predictive distribution : by using the above ,

prior x likelihood = posterior

• Conjugate Distribution : when we take the product of the distribution and its conjugate the result has the same form as the conjugate.
• Possible if you have conjugate prior.
• What do if you have non conjugate priors

Approximate Inference

Laplace Approximation

• Find a gaussian approximation q(z) which is centered on a mode of the distribution p(z)

Variational Inference

• Approximate intractable distribution p(theta given d) with simpler tractable distribution q(theta) by minimizing KL distane.
• Its an iterative procedure .

Monte Carlo Sampling

• Draw independent samples from distribution to approximate expectation, in some sense we take a weighted average.
• Can use importance sampling
• Monte Carlo methods are a class of techniques that use random sampling to simulate a draw from some distribution. By making repeated draws and calculating an aggregate on the distribution of those draws, it’s possible to approximate a solution to a problem that may be very hard to calculate directly.

Lecture 2

The basic VAE

• Bayesian meets Deep learning
• Autoencoders try to reconstruct data. The encoder network can be used for dimensionality reduction. The decoder "may" be used for generating images.
• VAEs are ultimately generative models.
• It's a latent variable model , so we need to learn a mapping from some latent variable z to a complicated distribtution on x. if we know z we can get g(z).
• We assume that the laten variable is simple.
• Leverage Neural networks to learn a latent variable model.
• Reparametrization trick , Allows VAE to backprop
• Look at Aaron Couville slides DLSS 2015 for math.

Applications

• Image and Video generation
• Super Resolution
• Forecasting from static images
• Everything GANs can be used for
• Attribute Conditioned Image Generation

VAEs end up generating blurry results because the formulation itself has an averaging effect. Issues : Component Collapse.

Other VAES

• Semi Supervised VAE
• Conditional VAE
• Importance Weighted VAE
• Denoising VAE
• Adversarial VAE

Lecture 3

Probabilistic Deep Learning : Advances

• Applying Probabilistic models in Deep learning
• Need for uncertainity , we need to know what the model doesnt know .
• Main idea : dropout training is identical to approximate inference in bayesian modeling
• By introducing dropout, you are essentially taking an expectation with respect to the weights.

Uncertainties that the model has

• Two types of probabilty : epistemic and Aleatoric
• Aleatoric has two types again
• use the dropout training directly, except while changing the loss function

Lecture 4

Recurrent VAEs, Attention and Extensions

• One of the ways to expand VAEs is recurrent VAEs, which has an attention mechanism.
• Unlike one shot image generation earlier, DRAW does it parts at a time.
• How it works is the first VAE cell tries to generate some part of the image then this part is subracted from the original image and is feeded into the next cell.
• There's a read and write mechanism. its an attention mechanism

Attention in deep learning

• RNNs for captioning : Divide the image into grid locations and let the network figure out what part it wants to focus on
• Two types of attention : soft{can use backprop} and hard attention{need rl}
• Spatial Transformer Network : We focus on a subpart of the image

DRAW

• Consists of RVAE with an attention mechanism
• Attention was used in a way that a point grid of 3x3 was to be placed on an image, then the network had to figure out where to put the centre of the image in and what should be the stride within these points while ensuring that it isnt out of bounds
• Also look at alignDRAW, which uses a bidirectional LSTM which learns a representation of the caption and feeds this into alignDRAW.

SyncDraw

• Extending R-VAE to videos
• One RVAE for entire video, but a different attention mechanism for each frame.
• there's also syncdraw with captions, Conditioning on captions produces better results
• Video generation using dual attention, soft attention for long term and short term proposed.